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ABSTRACT 


Artificial  intelligence  (“intelligent  agent”)  technology  is  widely  deployed  in 
numerous  commercial  areas  such  as  networking,  modeling,  and  software;  however,  this 
technology  remains  under-utilized  by  operational  organizations  within  the  United  States 
Navy.  This  thesis  will  investigate  the  importance  of  artificial  intelligence  (AI)  for 
military  training  simulations,  particularly  in  the  training  of  intelligence  personnel  in  the 
Navy. 

The  Computer  Generated  Forces  (CGF)  of  the  current  Intelligence  Team  Trainer’s 
(ITT)  system  initiate  actions  as  a  result  of  rigid  scripted  programming.  Forces  will 
execute  the  same  actions  regardless  of  what  the  user  decides  to  do,  resulting  in  highly 
unrealistic  scenarios.  For  instance,  in  a  scenario  where  an  ARG  (Amphibious  Ready 
Group)  transits  the  Strait  of  Hormuz,  the  response  of  Iranian  P3  or  an  incoming  dhow 
would  be  the  same  whether  the  battle  group  utilized  frigate  escorts  or  not. 

This  thesis  will  produce  very  simple,  but  less  rigid  AI,  which  can  easily  be  made 
more  complex  and  “intelligent”  in  later  phases.  Demonstrations  and  assessments  will 
validate  the  importance  of  AI  integration  for  the  ITT.  Furthermore,  this  analysis  of  the 
requirements  for  the  AI  will  assist  training  commands  and  combat  information  centers 
fleet  wide  with  the  range  of  realistic  combat-related  possibilities  needed  in  order  to 
ensure  a  fully  capable  ‘combat  ready’  watch  team. 
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I. 


INTRODUCTION 


A.  PROBLEM 

In  the  Department’s  Transformation  Planning  Guidance,  Secretary  of  Defense 
Rumsfeld  stated,  “We  must  transform  not  only  the  capabilities  at  our  disposal,  but  also 
the  way  we  think,  the  way  we  train,  the  way  we  exercise,  and  the  way  we  fight.” 
(Training04)  Within  this  document  we  focus  on  how  ‘training’  within  the  military  and 
other  internal  components  within  the  DoD  must  now  prepare  the  force  to  learn, 
improvise,  and  adapt  to  constantly  changing  threats  while  simultaneously  executing 
doctrine  to  standards.  While  it  is  obvious  that  training  of  some  kind  is  essential  in  any 
strategic  environment,  the  method  of  approach  is  just  as  vital.  That  is  where  the  use  of 
computer  simulation  training  plays  a  major  role  in  the  military  today.  More  and  more 
computer-based  training  is  being  introduced  into  the  military  as  a  means  to  not  only  save 
money,  but  to  allow  capabilities  that  would  otherwise  be  limited  to  the  real-world  and 
access  to  more  than  one  individual  at  a  time. 

Despite  many  advances  achieved  within  the  fields  of  Modeling  and  Simulation 
(M  &  S)  and  agent  technology  over  the  past  several  decades,  practical  application  of  both 
have  been  highly  under  utilized  by  the  operational  units  of  the  United  States  Navy.  When 
such  technology  has  been  deployed  with  units,  it  has  usually  been  developed  in  order  to 
improve  operator  proficiency,  which  is  also  the  focus  of  this  work.  Little  has  been 
created  for  the  war  fighter  to  run  “what-if  ’  scenarios  that  might  aid  in  development  of 
tactical  planning  for  increased  proficiency  of  established  doctrine. 

B.  MOTIVATION 

Currently,  battle  group  (BG)  intelligence  teams  are  required  to  spend  a  minimum 
of  three  training  sessions  at  the  Intelligence  Team  Trainer  (ITT)  at  the  Fleet  Intelligence 
Training  Center,  Pacific  (FITCPAC).  This  trainer  is  a  highly  realistic  simulation  of  the 
various  intelligence  cells  onboard  ships  and  trains  the  teams  within  the  different  cells  as 
well  as  coordination  between  the  multiple  cells  of  the  BG.  Resembling  many  trainers 
currently  employed  fleet  wide,  the  trainer  is  not  meant  to  train  individuals  about  their 
jobs,  for  job  competency  is  trained  earlier  and  is  assumed  by  the  time  the  team  reaches 
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FITCPAC.  The  trainer  is  utilized  as  a  way  to  teach  the  team  to  function  proficiently  and 
that  a  team’s  value  is  the  sum  of  all  of  its  parts. 

Within  the  ITT,  operators  sit  at  mock  consoles  that  replicate  actual  seating 
arrangements  that  are  utilized  on  deployment  and  all  input  into  these  systems  are  highly 
accurate.  Accurate  message  traffic,  including  some  that  do  not  relate  to  the  problem,  is 
distributed  though  out  the  cells.  See  Figure  1  below. 


SUPP LOT  Team  Trainer  Stations: 


Figure  1.  SUPPLOT  Team  Trainer  Stations  (From:  MOVES  Institute  Proposal) 


As  intelligence  analysts,  the  team  members  interpret  and  use  this  information  to 
make  a  determination  on  the  operation  situation  that  is  then  passed  on  to  BG  commanders 
via  their  Watch  Officer.  Due  to  the  substance  of  the  scenarios,  the  details  of  each  are 
tightly  scripted  and  take  a  significant  amount  of  time  to  design  and  build. 

The  amount  of  time  it  takes  to  create  each  scenario  is  one  of  many  obstacles  to  the 
ITT.  Currently,  each  scenario  takes  up  to  one  year  to  build  and  as  a  result  few  scenarios 
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exist  which  reflect  the  latest  conditions.  This  has  resulted  in  each  BG  intelligence  team 
conducting  the  same  scenario  multiple  times.  Without  unpredictability  and  variation,  the 
team’s  performance  becomes  less  that  of  an  ability  of  a  team  to  function  properly,  but 
rather  their  ability  to  remember  each  scenario.  Additionally,  there  is  no  way  for  the  BG 
intelligence  team’s  to  train  on  scenarios  based  on  the  latest  changes  in  the  tactical, 
operational,  or  strategic  situation. 

C.  THESIS  STATEMENT 

This  thesis  describes  the  investigation,  research,  and  development  of  a  less  rigid 
artificial  intelligence  (AI)  which  can  easily  be  made  more  complex  and  “intelligent”  in 
later  phases.  Demonstrations  and  assessments  validate  the  importance  of  AI  integration 
for  the  ITT.  Furthermore,  this  analysis  of  the  requirements  for  the  AI  will  assist  training 
commands  and  combat  information  centers  fleet  wide  with  the  range  of  realistic  combat- 
related  possibilities  needed  in  order  to  ensure  a  fully  capable  ‘combat  ready’  watch  team. 
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II.  BACKGROUND  INFORMATION 


A.  INTRODUCTION 

Due  to  the  requirements  of  their  work,  intelligence  professionals  are  not  always 
given  the  opportunity  to  train  as  a  team  prior  to  a  deployment.  A  response  to  this  need 
was  developing  an  Intelligence  Team  Trainer  supervised  by  military  instructors  and 
civilian  contractors  that  allowed  these  teams  to  not  only  meet  fleet  requirements  for 
inspections,  but  to  allow  them  to  train  in  a  manner  similar  to  their  deployed  environment. 
The  trainer  establishes  rigorous  skill  sets  that  build  upon  one  another,  so  the  end  result  is 
an  effective  and  efficient  intelligence  operation  that  can  easily  communicate  with  other 
intelligence  operations  and  teams  throughout  not  only  their  own  Battle  Group,  but 
through  out  the  world’s  intelligence  cohorts. 

The  intended  end  state  of  each  training  simulation  is  to  expose  the  teams  to 
situations  that  they  will  encounter  while  underway.  In  other  words,  the  trainer’s  intention 
is  to  better  prepare  a  sailor,  allowing  them  have  a  better  grasp  on  what  to  do  and  how  to 
react  in  a  timely  manner  for  various  events  they  will  encounter  in  the  real  world. 
Additionally,  a  more  general  goal  of  this  trainer  is  to  give  the  Sailors  the  opportunity  to 
learn  how  to  work  together  as  a  team  in  an  Operational  setting. 

B.  GLOBAL  COMMAND  AND  CONTROL  SYSTEM  (GCCS) 

The  Global  Command  and  Control  System  (GCCS)  is  an  automated  information 
system  designed  to  support  a  common  operation  picture  with  the  use  of  an  integrated  set 
of  analytic  tools  and  the  flexible  data  transfer  capabilities.  GCCS  is  fast  becoming  the 
single  C4I  system  to  support  the  warfighter  from  foxhole  to  unified  command  posts 
(GCCSOO). 

The  purpose  of  GCCS  is  to  provide  command  centers  with  C5I,  maintain  the 
common  operating  picture  (COP)  for  the  specified  area  of  responsibility  (AOR),  and 
provide  the  DoD  with  an  open  system  architecture,  as  shown  in  Figure  2. 
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Figure  2.  Illustration  of  GCCS  Elements  (From: 
http://www.fas.org/nuke/guide/usa/c3i/gccsl  .gif ) 


GCCS  is  known  to  most  in  the  military  as  the  god’s  eye  view  of  the  world.  It 
enables  trained  personnel  to  keep  track  of  units  (US,  friendly,  and  hostile)  throughout  the 
world  on  one  screen  with  tools  that  allow  for  modifications  and  transfering  data  to  and 
from  databases  maintained  by  the  Defense  Intelligence  Agency  (DIA). 

There  are  four  capabilities  and  operations  of  GCCS: 

1 .  Databases 

•  National,  Tactical,  and  Local  data 

•  Intelligence  Databases  (Modernized  Integrated  Database  (MIDB), 
populations,  units,  facilities) 

2.  Imagery 

•  If  stored,  can  be  manipulated 


JIVE  is  the  MIDB  and  image  link  program  (PC  side) 

6 


•  Good  to  use  C2PC  or  Intel  Office  to  maintain 

3.  Message  Handling  and  Communications  Processing 

•  Lines  of  Communication  (LOC’s)  and  IR’s 

4.  Track  Correlation  for  near  real-time  update 

•  For  Force  Over-the-Horizon  Track  Coordinator  (FOTC) 

•  Locator  data 

•  Longitude  and  Latitude 

•  Link  data  (1 1/14/16) 

In  order  to  operate  this  system  deployed  or  land-based,  manning  of  GCCS 
involves  two  main  categories  of  personnel:  users  and  maintainers. 

Users 

1.  Intelligence  Specialist  (IS)  -  Tactical  /  Strategic  Intelligence 

2.  Operations  Specialist  (OS)  -  FOTC 

3.  Electronic  Warfare  (EW)  -  Maintain  radars 

4.  Cryptologic  Technician  (CT)  -  ELINT  correlation  and  database 

Maintainers 

1.  Electronic  Technicians  (ET)  -  Maintain  hardware 

2.  Information  Systems  Technician  (IT)/  Cryptologic  Technician  - 
Communications  (CTO)  -  software  and  application 

The  users  and  maintainers  of  the  system  are  Navy  enlisted  personnel  with 

subspecialties  that  qualify  them  to  train  for  this  system.  In  general,  each  rate  has 

different  positions  within  the  trainer  and  requires  a  prerequisite  such  as  a  special  school 

or  qualification  in  order  to  do  their  job.  For  instance,  the  Intelligence  Specialist  (IS),  if 

attached  to  a  ship,  must  attend  the  Global  Command  and  Control  System  Maritime  Fleet- 

School  (GCCS-M  F)  held  at  the  FITCPAC  center  in  San  Diego  (for  West  Coast 

personnel).  Once  they  graduate  from  the  school  they  receive  a  certification  that  qualifies 

them  to  operate  the  GCCS  system  and  any  application  related  to  the  system. 

Additionally,  each  piece  of  equipment  or  application  in  the  system  requires  a  certification 

from  its  associated  training  environment. 

GCCS  is  the  principal  command  and  control  system  supporting  the  Joint  Chiefs  of 
Staff  (JCS)  and  Commanders  in  Chief  (CINCs)  in  managing  military  assets.  GCCS 
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support  six  mission  areas  (operations,  mobilization,  deployment,  employment, 
sustainment,  and  intelligence)  through  eight  functional  areas: 

•  Threat  identification  and  assessment 

•  Strategy  planning  aid 

•  Course  of  action  development 

•  Execution  planning 

•  Implementation 

•  Monitoring 

•  Risk  analysis 

•  Common  tactical  picture 

C.  MODERNIZED  INTEGRATED  DATABASE  (MIDB) 

The  Modernized  Integrated  Database  (MIDB)  is  considered  the  worldwide 
general  military  intelligence  (GMI)  database  for  the  Distributed  Production  Program 
(DPP)  to  provide  GMI  intelligence  to  the  warfighter.  MIDB  is  currently  available 
through  Intelink,  the  Sensitive  Compartmentalized  Information  network.  MIDB  provides 
access  in  several  different  forms  such  as  static  posting,  templated  dynamic  pull,  and  full 
MIDB  access  via  Telnet.  MIDB  serves  as  the  principal  database  of  intelligence  data  for 
the  entire  United  States  Department  of  Defense  (DOD)  community,  as  well  as  Australia, 
Canada,  and  the  United  Kingdom.  MIDB  Order  of  Battle  data  contains  all  official 
validated  graphical  and  textual  information  about  enemy  sites,  facilities,  and  units.  The 
system  supports  MIIDS/IDB  database  (military  forces,  installation  and  facilities, 
population  concentrations,  command  and  control  structures,  significant  events  and 
equipment)  for  retrieval  and  maintenance.  MIDB  is  the  MUDS  database  with  the 
addition  of  icons  that  represent  the  data  in  the  database,  as  shown  in  Figure  3.  Data 
collected  from  this  database  is  extremely  important  to  the  intelligence  team  trainer  and  is 
pulled  on  a  daily  basis  to  maintain  an  updated  operational  picture  of  the  world  and,  more 
importantly,  the  area  of  operation. 
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Figure  3.  Mainframe  IDB  (From: 
http://www.fas.org/irp/program/disseminate/midb_build.gif) 


MIDB  is  a  DOD  migration  system  and  currently  includes  several  legacy  systems: 

•  Electronic  Order  of  Battle  Services  (EOBS) 

•  Expeditionary  Warfare 

•  Military  Facilities  File  (MILFAC) 

•  Target  Material  Management  (TMM) 

•  CENTCOM/SOCOM  Integrated  Data  System  (CSIDS) 

•  Force  Trends  database  (FORT) 

•  Force  Tracking  Information  System  (FORTRIS) 

•  Space  Data  Base  (SDB) 

D.  ARTIFICIAL  INTELLIGENCE 

There  has  been  a  revolution  of  change  in  game  design,  slowly  evolving  from  a 
linear  aspect,  such  as  scripted  game  play  to  an  environment  of  exponential 
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unpredictability  (Tozour03).  These  changes  have  brought  about  numerous  approaches 
and  techniques  relevant  to  applying  variability  to  responding  to  a  user’s  actions  in  order 
to  create  a  more  realistic  training  environment.  Among  these  techniques  are  genetic  and 
heuristic  algorithms,  scripting,  state  machines,  A*  Path  finding,  and  Bayesian  networks. 
According  to  Paul  Tozour,  the  most  useful  of  these  in  creating  a  more  human-like 
reasoning  is  the  probabilistic  inference  technique  of  Bayesian  networks  (Touzour02_l, 
Touzour02_2).  Additional  contributions  have  been  made  by  the  entertainment  sector, 
and  due  to  advancements  in  hardware  and  software  continuously  become  more  accessible 
to  the  everyday  user. 

For  decades  the  military  has  built  computerized  tools  for  analyzing  combat  such 
as,  ATLAS,  CASTFOREM,  THUNDER,  and  T  AC  WAR,  to  name  a  few.  Recently, 
simulations  have  taken  over  as  the  foundation  for  computerized  combat  training  tools 
(Janus,  AWSIM,  and  JTLS).  The  expense  of  producing  these  computerized  training  tools 
is  phenomenal.  Realizing  the  cost  involved  in  production  and  the  continuing  influence  of 
producing  accurate  training  systems,  the  Department  of  Defense  (DoD)  made  the 
validation,  verification,  and  accreditation  (VV&A)  of  these  simulations  an  official 
requirement  (DOD5000,  Training04). 

The  difficulty  in  conducting  VV&A  for  a  particular  system  is  daunting  and 
requires  individuals  with  a  great  deal  of  experience  in  building  and  utilizing  military 
simulations.  Numerous  papers  have  been  written  in  this  area;  however,  little  is  written  on 
the  effects  of  systems  with  rigid  programming  (i.e.,  scripted  events)  versus  systems 
utilizing  AI.  In  an  article  written  by  John  E.  Laird,  he  studies  the  similarities  and 
differences  in  AI  for  computer  games  and  AI  utilized  in  computer  generated  forces. 
(LairdOO) 

In  most  legacy  military  training  tools,  the  framework  behind  the  system  exploits 
scripted  events.  In  other  words,  enemy  actions  behave  the  same  every  time  the 
simulation  is  run.  When  a  validation  of  the  effects  of  these  types  of  systems  on  trainee 
performance  is  conducted  over  a  sustained  period  of  time,  performance  levels  of 
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individuals/teams  continually  increase.  Of  course,  it  is  painfully  obvious  why:  the 
individual/team  knows  exactly  what  the  enemy  will  do  each  time.  But  this  cannot  be 
called  effective  training. 

Many  examples  of  practical  approaches  to  applying  AI  to  game-based  training  are 
available  to  the  military.  Among  them  is  an  article  by  John  Manslow  who  discusses  the 
importance  of  the  application  of  using  AI  in  order  to  force  players  to  continually  search 
for  new  ways  to  defeat  the  enemy  AI.  He  provides  solutions  and  steps  in  the  decision 
processes  of  making  an  adaptive  AI.  (Manslow02). 

The  objective  of  many  training  systems  is  enhanced  training  value  with  no 
negative  training.  Detail  or  the  ‘fidelity’  of  a  training  system  is  dependent  on  the  goal  of 
the  trainer.  There  are  numerous  papers  that  discuss  training  transfer  or  increasing  or 
decreasing  fidelity  based  on  the  objective  of  the  simulation  or  virtual  environment.  For 
this  reason,  this  paper  will  focus  on  performance  measures  and  explaining  how  to  obtain 
them  in  a  system  that  utilizes  very  simple  AI.  The  metric  obtained  in  this  analysis  will 
demonstrate  the  superiority  of  using  simple  AI  rather  than  none  at  all  in  combat  training 
tools. 
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III.  REQUIREMENTS  ANALYSIS 


A.  EVALUATING  TRAINING  GOALS 

During  the  researcher’s  visit  to  the  Intelligence  Team  Trainer  (ITT),  instructors 
discussed  their  desired  functionality  in  a  new  training  system.  The  instructors  expressed 
great  interest  in  the  use  of  AI  in  order  to  allow  them  to  utilize  time  spent  in  the  trainer  to 
overlook  the  team  itself,  instead  of  typing  message  traffic  or  equivalent  activities  that 
take  away  from  complete  observation  of  the  team.  Additionally,  the  instructors  desired 
pull-down  menus  to  set-up  various  scenarios  based  on  team  experience  and  time 
limitations.  The  pull-down  menus  include  a  range  of  selections  from  areas  of 
responsibility  (AORs),  time  of  day,  weather,  countries  involved,  etc.  This  will  be 
covered  in  more  detail  in  the  Design  Chapter. 

In  a  meeting  conducting  between  FITCPAC  personnel  and  the  MOVES  Institute 
at  Naval  Postgraduate  School,  the  following  critical  factors  for  a  successful  game-based 
simulation  were  discussed.  The  trainer  should  have  all  the  basic  functionality  of  the  old 
system  in  addition  to  the  following: 

•  The  trainer  should  be  simple  enough  to  use  so  that  both  staff  at  FITCPAC 
and  deployed  personnel  can  utilize  the  trainer  and  produce  scenarios 
within  without  special  training; 

•  The  trainer  should  be  PC  based  in  order  for  personnel  to  use  the 
simulation  exclusive  of  any  access  to  specialized  systems  which  might  be 
occupied  by  a  team  or  inoperative  do  to  scenario  setup.  The  specialized 
systems  would  include  the  MIDB,  GCCS,  and  Office  Intel; 

•  The  system  should  be  capable  of  training  a  complete  intelligence  team, 
requiring  a  network  that  allows  team  members  to  interface  with  one 
another; 

•  The  system  should  be  capable  of  training  a  single  operator,  with  the  other 
team  members  simulated  by  agents  that  are  indistinguishable  from  actual 
humans.  For  instance,  if  one  user  desires  to  operate  a  PC  based  version  of 
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the  trainer,  the  individual  stations  not  manned  will  be  simulated  by  the 
game  with  responses  and  actions  expected  from  an  actual  qualified  watch 
stander; 

•  The  trainer  should  be  capable  of  interfacing  and  stimulating  the  systems 
and  equipment  involved  in  normal  use  by  the  intelligence  team  in  order  to 
create  the  same  environment  that  is  takes  place  while  deployed; 

•  The  trainer  should  be  reactive,  in  order  that  the  user’s  actions  produce  the 
enemy’s  actions.  The  system  will  also  permit  the  enemy  to  “learn”  from 
the  user’s  actions  so  that  the  same  user  action  will  not  produce  identical 
results  each  time; 

•  The  trainer  should  be  entertaining  and  fun,  so  that  Sailors  will  play  the 
simulation  outside  of  the  work  environment  allowing  for  maximization  of 
training. 

B.  THESIS  SCOPE  LIMITATIONS 

Due  to  time  and  capability  limitations,  this  thesis  provides  the  basic  functionality 
of  the  current  system’s  capabilities  limiting  itself  to  one  scenario.  Additionally,  this 
thesis  provides  a  simple  PC  based  version  of  the  trainer  with  a  scenario  generator,  simple 
MIDB,  message  traffic  implementation,  and  a  graphical  user  interface  that  displays  the 
actions  of  the  user  and  the  enemy’s  response  to  that  action.  This  thesis  focuses  only  on 
the  issue  of  the  reactivity  of  the  trainer  to  user  actions.  However,  one  scenario  will  be 
sufficient  enough  to  provide  evidence  whether  utilizing  artificial  intelligence  is  an 
improvement  over  the  older  scripted  version. 

C.  CURRENT  SYSTEM  FUNCTIONALITY 

The  ITT  utilizes  the  framework  from  GCCS,  MIDB,  Office  Intel,  and  GCCS  is 
the  main  operating  system  for  the  ITT  simulation  at  FITCPAC,  San  Diego.  Use  of  a 
modified  stand-alone  version  is  maintained  at  the  FITCPAC  facility  to  facilitate  the 
realistic-scenario  driven  simulations  conducted  by  the  military  instructors  without 
disturbance  to  the  real  world. 

Scenarios  are  put  together  by  contractors  with  experience  in  scripting,  doctrine, 
and  various  applications  utilized  in  order  to  create  realistic  events.  The  process  of 
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creating  these  scenarios  is  very  tedious  and  takes  two  qualified  contractors  approximately 
a  year  to  build.  First,  research  into  doctrine  of  the  countries  involved  must  be  conducted 
so  as  to  script  rational  results.  Secondly,  orders  of  battle  must  be  loaded  in  from  the 
stand-alone  MIDB  that  include  all  countries  involved  which  are  placed  in  specific  areas 
according  to  the  research  conducted  by  the  contractors.  Thirdly,  event-driven  scenarios 
are  scripted  one  enemy  reaction  at  a  time  dependent  on  the  action  of  the  user. 

The  current  trainer  progresses  through  the  interaction  of  the  watch  standers  and 
instructors  based  on  the  decisions  made  by  the  watch  standers  using  data  provided  for 
them.  The  watch  standers  interact  through  chat  and  analyzing  message  traffic  and  visual 
data  provided  for  them  by  display.  The  watch  standers  produce  a  complete  intelligence 
picture  for  the  Force  Intelligence  Watch  Officer  (FIWO),  who  in  turn  makes 
recommendations  to  “the  staff’,  which  is  simulated  by  the  FITCPAC  instructors  who  are 
observing  the  scenario.  The  team’s  recommendations  are  always  taken  by  the  staff  and 
the  system  runs  its  scripted  actions. 

D.  SYSTEM  DEFICITS 

The  current  system’s  forces  initiate  actions  as  a  result  of  scripted  programming. 
In  other  words,  forces  will  execute  the  same  actions  regardless  of  what  the  user  decides  to 
do,  resulting  in  highly  unrealistic  scenarios.  For  instance,  in  a  scenario  where  an  ARG 
transits  the  Strait  of  Hormuz,  the  response  of  Iranian  P3  or  an  incoming  dhow  would  be 
the  same  whether  the  battle  group  utilized  frigate  escorts  or  not.  Because  of  this,  scripted 
events  are  generally  inferior  to  AI  for  the  overall  objective  of  FITCPAC ’s  Intelligence 
Team  Trainer. 

Due  to  the  amount  of  time  it  takes  to  create  a  complete  scenario,  the  FITCPAC 
facility  will  have  only  two  or  three  scenarios  available  to  them  per  training  team  cycle. 
This  results  in  the  intelligence  team  running  each  scenario  approximately  three  times, 
each  time  with  same  situation  and  same  behavior  of  enemy  forces.  Scripted  events  limits 
the  amount  of  reaction  an  enemy  force  will  have  as  a  result  of  the  user’s  action.  The 
team  can  become  lethargic,  simply  because  if  they  are  familiar  with  the  scenario,  they  can 
repeat  previous  actions  with  complete  knowledge  of  what  the  enemy  will  do.  The 
training  becomes  monotonous,  and  users  will  continue  on  with  the  scenario  without 
reaching  their  full  strategic  potential. 
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E.  POSSIBLE  SYSTEM  IMPROVEMENTS 

The  best  way  to  improve  this  training  is  to  use  AI  to  create  enemy  forces  (often 
referred  to  as  “red  forces”)  and  U.S.  forces  (“blue  forces”)  which  will  react  to  one 
another.  Therefore,  each  time  the  same  simulation  set-up  is  rendered,  the  behavior  of  the 
enemy  forces  will  not  necessarily  be  the  same,  since  the  actions  the  trainees  take  will 
change  the  behavior  of  the  blue  forces  which  will  effect  the  behavior  of  the  red  forces. 
Furthermore,  it  is  possible  to  introduce  variation/randomness  in  the  behavior  of  the 
enemy  agents  even  given  identical  actions  by  the  user.  Adding  AI  like  to  the  current 
system  will  improve  the  realism  of  scenarios  for  the  training  facility  at  FITCPAC. 
Additionally,  such  a  system  will  be  deployable  with  each  shipboard  intelligence  team. 
Building  a  system  to  introduce  such  variability  in  force  behavior  is  the  aspect  we  focus 
on  in  this  work. 

Another  advantage  of  utilizing  AI  within  the  current  system  is  that  it  will 
significantly  reduce  the  time  spent  on  creating  scenarios.  As  mentioned  earlier,  it  takes 
approximately  a  year  to  generate  a  scripted  scenario.  Adding  pull-down  menus  to 
establish  each  scenario  prior  to  running  it  will  significantly  shorten  time  spent  on 
creation.  Of  course,  creating  a  system  that  enables  a  scenario  to  be  drafted  regardless  of 
what  country  and  area  of  responsibility  will  require  a  significant  amount  of  research. 
This  thesis  covers  only  one  scenario,  involving  the  ongoing  tensions  with  the  arms  race 
of  China  and  its  presumed  desire  of  Taiwan  via  a  situation  involving  the  Straight  of 
Taiwan.  Details  will  be  discussed  later  in  Chapter  IV. 

Additionally,  by  generating  message  traffic  with  the  AI  the  instructors  will  have 
the  opportunity  to  be  continuously  present  in  the  trainer  with  the  team,  vice  at  modules 
typing  up  responses.  This  will  produce  a  better  training  environment  because  the 
instructors  will  have  more  time  to  critique  and  mentor  the  students.  The  system  will  also 
have  the  capability  to  record,  stop,  and  replay  a  scenario  at  the  instructors’  preference. 
This  functionality  will  allow  for  the  optimum  training  environment  for  the  instructors  to 
discuss  with  the  team  their  decisions  and  how  to  improve  their  decision  making  process. 
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IV.  SOFTWARE  PROTOTYPE 


A.  INTRODUCTION 

This  chapter  will  provide  an  overview  of  the  software  developed  to  support  the 
experimental  investigation  described  later. 

B.  SYSTEM  ARCHITECTURE 

The  game-based  simulator  was  created  using  NetBeans,  which  is  an  open  source 
integrated  development  environment  (IDE)  written  entirely  in  Java  using  the  NetBeans 
Platform.  NetBeans  is  based  upon  the  Java  language  which  contains  the  ability  to 
provide  simple  graphical  user  interface  (GUI),  making  it  easier  to  develop  complex 
behavior.  Java  contains  modules  to  handle  networking,  rendering,  movement,  and 
complex  interactions  between  objects.  The  Java  code  will  act  like  a  game  engine  as 
shown  in  the  architecture  diagram,  Figure  4. 


Figure  4.  Architecture  Diagram 


The  program  was  written  utilizing  various  classes  that  control  different  aspects  of 
the  game-based  simulator.  Each  class  has  a  specific  purpose  within  the  simulation  such 
as  setup  of  initial  variables,  generating  message  traffic,  movement  of  forces,  and 
generation  of  the  GUI. 
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The  initial  architecture  designed  for  this  thesis  is  limited  to  just  the  scope  of  the 
thesis.  The  application  demonstrates  for  the  user  the  distinction  between  a  fixed-result 
simulation  and  a  simulation  developed  with  AI. 

C.  FEATURES 

In  order  to  create  a  miniaturized  stand-alone  version  of  the  current  system,  steps 
were  taken  so  that  all  applications  utilized  in  this  test  platform  are  as  similar  as  possible 
to  the  real  system.  Miniaturized  versions  of  systems  include:  the  GUI,  MIDB,  Message 
Traffic  Generation,  and  an  addition  to  the  current  system  which  is  the  pull-down  scenario 
creator  (  See  Figure  5). 

1.  Graphical  User  Interface 

All  the  features  involved  within  the  simulation  are  conducted  in  a  GUI  built 
within  the  NetBeans  IDE.  The  opening  display  allows  the  user  to  setup  the  simulation 
from  a  series  of  buttons  that  open  up  separate  windows  with  allow  users  to  choose 
different  preferences  on  how  they  would  like  to  run  the  simulation.  This  allows  the  user 
to  select  area  of  operation  (AO),  time  of  day,  and  intensity  level.  Each  selection 
determines  the  variables  involved  within  each  class  and  function  that  help  govern  the 
course  of  the  simulation.  A  brief  description  of  each  class  and  its  purpose  within  the 
simulation  is  discussed  later  in  this  chapter  under  Implementation. 
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Figure  5.  Screen  Shot  1  of  Simulation 


2.  Modernized  Integrated  Database  (MIDB)  Structure 

First,  research  was  conducted  in  order  to  analyze  China’s  military  doctrine  and 
history  of  reactions  to  various  combat  situations.  Additionally,  Taiwan  was  studied  in 
order  to  establish  its  military  standing  and  social  dependency  on  China.  A  very  simple 
order  of  battle  was  established  for  China,  Taiwan,  Britain,  Australia,  Japan,  and  the 
United  States.  Assumptions  of  course  were  made  that  all  parties  would  be  involved  on 
the  influence  of  the  United  Nations  and  previous  experience  with  established  coalitions 
during  Operation  Enduring  Freedom.  The  original  MIDB  would  provide  a  substantial 
amount  of  intelligence  data,  but  for  the  purposes  of  this  thesis,  data  is  limited  to  only  data 
needed  to  conduct  this  experiment. 

The  MIDB  be  used  in  this  thesis  will  be  based  on  a  plain  text  file  built  via  the 
very  simple  and  well  known  text  editor  Notepad.  In  future  versions,  Microsoft  Excel  or 
Intel  Excel  should  be  used  to  simplify  the  entry  process.  Notepad  allows  for  simple 
extraction  of  the  data  needed  to  establish  order  of  battle  for  red  and  blue  forces.  The  data 
extracted  will  include  the  following: 


19 


•  Military  bases  and  facilities  (i.e.  port,  airbases,  SAM  sites) 

•  Military  units  (i.e.  types  of  ships,  aircrafts) 

•  Weapons  associated  with  each  platform 

•  WMD 

3.  Message  Traffic  Generation 

Intelligence  information  gathered  from  and  transmitted  via  message  traffic  is 
crucial  for  the  BG  intelligence  teams.  Messages  received  via  the  Naval  Computer  and 
Telecommunications  System  (NCTS)  are  transmitted  and  disbursed  throughout  all  U.S. 
Naval  Ships  and  Shore  Facilities.  Information  contained  within  each  message  must 
follow  a  specific  format  encompassing  subject  matter  that  is  classified  as  unclassified, 
confidential,  secret,  or  top  secret.  Format  of  messages  are  in  accordance  with  Naval 
Telecommunications  Procedure  (NTP3(J)97)  and  for  the  purposes  of  this  thesis,  all 
messages’  will  be  labeled  as  confidential;  however,  no  actual  confidential  information 
will  be  created  or  used  (Figure  7,  Figure  8). 


'  Intelligence  Reports 


Q  Messages  Chat  Contacts 


9  C3  Messages 


Q  210001Z 


ZNYCCCCC 
P  21 0001Z  MAR  06 

FM  N  AVLANTF ITC  P  AC  NORFOLK  VA//N30// 

TO  USS  SHIPtfJJJtf 

INFO  CINCLANTFLT  NORFOLK  VMN5/1 
BT 

C  0  N  F  I  D  E  N  T I  A  L//N02300/1 

NEWS  REPORTS  THAT  TAIWAN  ABANDON'S  "ONE-CHINA" 
PRINCIPLE  BUT  DOES  NOT  DECLARE  INDEPENDENCE. 
CHINA  REMOVES  RIGHTS  FOR  US  NAVAL  SHIP  PORT 
CALLS. 


Figure  6.  Message  Traffic  Generator  Initial  Message 
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Figure  7.  Message  Traffic  Generator 

The  purpose  of  the  message  traffic  generation  within  the  thesis  scope  is  strictly 
confined  to  demonstration  of  the  established  scenario  in  the  thesis  simulation.  In  future 
work,  AI  should  be  utilized  to  generate  messages  as  deemed  necessary  by  user’s  actions. 
Messages  in  this  simulation  however,  are  messages  that  are  pre-written  and  will  emerge 
within  the  simulation  at  pre-determined  time  increments.  For  example,  at  time  x, 
message  02  will  appear;  at  time  x  +1,  message  03  will;  appear  and  so  forth. 

D.  IMPLEMENTATION 

This  section  will  describe  briefly  the  structure  of  classes  involved  in  the 
demonstration  of  this  thesis. 

1.  Selected  Classes  and  Methods 

The  simulation  uses  the  following  classes  and  methods  and  a  concise  summary  of 
each  is  shown: 
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Settings  class.  This  class  is  the  initial  display  which  allows  the  user  to  choose 
various  options  available  to  determine  the  variables  involved  in  the  classes  listed  below 
to  run  the  simulation.  For  example,  time  of  day,  area  of  operation,  and  intensity  of  the 
scenario. 

Start  class.  This  class  runs  the  GUI  to  get  instructor  inputs  that  determine  what 
happens  within  each  time  dependent  portion  of  the  simulation.  Within  the  GUI  there  is 
the  chart  of  the  area  of  operation  (in  this  case  Strait  of  Taiwan),  controls  that  stop  and 
start  the  simulation,  and  the  display  that  allows  the  user  to  scroll  through  message  traffic 
available  to  him/her  at  various  times  throughout  the  demonstration.  See  Figure  5. 


Figure  8.  Screen  Shot  of  simulation 


Coordinate  class.  This  class  was  written  to  assist  in  icon  placement  and 
movement  throughout  the  2D  environment  on  the  chart.  Movement  is  dependent  on  the 
speed  of  the  icon  (i.e.  ship,  aircraft,  or  static  shore  facility)  which  is  shown  by  the  speed 
vector  in  the  direction  of  movement. 
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MessageTree  class.  This  class  provides  the  message  generation  and  message 
distribution  of  the  simulation. 

SimElement  class.  This  class  is  an  abstract  class  which  has  methods  and 
functions  that  are  similar  to  all  aircraft,  surface  vessels,  and  submarines,  (i.e.,  speed, 
quantity,  location).  Classes  that  implement  this  interface  are  those  written  to  control  each 
subclass  such  as  AircraftElement,  ShipElement,  and  so  forth.  This  class  was  written  in 
order  to  ease  the  complication  of  creating  a  list  of  objects  and  allowing  easy  movement  of 
each  regardless  of  the  entity. 
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V.  TEST  SCENARIO 


A.  INTRODUCTION 

This  chapter  will  provide  a  brief  summary  of  the  history  and  politics  behind  the 
chosen  scenario  of  the  Strait  of  Taiwan  is  given.  Additionally,  this  chapter  will 
summarize  the  initial  system  demonstration  setup  and  parameters  involved  in  the  pilot 
experiment  that  was  conducted  in  order  to  demonstrate  the  importance  of  exploiting  AI 
technology  in  military  training  systems. 

1.  Background  Information  on  Scenario 

Since  the  presidential  elections  of  March  2000,  the  world’s  attention  is  once  again 
focused  on  Taiwan.  There  has  been  an  ongoing  and  increasing  concern  over  the  island’s 
security,  in  particular,  whether  not  China  will  use  force  against  Taiwan  and  of  course, 
whether  or  not  the  U.S.  will  intervene  to  help  defend  the  small  island  of  Taiwan. 

The  rapid  rise  of  the  Peoples  Republic  of  China  (PRC)  as  a  regional  political  and 
economic  power  with  global  aspirations  is  one  of  the  principle  elements  of  the  emergence 
of  East  Asia.  As  a  result  of  China’s  military  expansion  and  the  expressed  political  desires 
of  the  concept  of  “one-China”,  the  focus  of  the  PRC  is  clearly  the  prevention  of  Taiwan 
independence.  (Studeman98)  Independence  has  not  been  declared  by  Taiwan,  but  despite 
negotiations  that  been  ongoing  between  both  parties,  Taiwan  has  made  it  known  to  the 
world  that  they  will  never  settle  on  Beijing’s  current  terms. 

The  likelihood  of  a  conflict  with  China  as  a  consequence  of  force  against  Taiwan 
is  a  realistic  possibility.  Therefore,  a  scenario  based  on  this  exact  conflict  involving  the 
prospect  of  U.S.  intervention  with  variables  that  would  play  influential  roles  in  decision 
making  is  ideal  for  this  investigation  of  AI. 

B.  INITIAL  SYSTEM  DEMONSTRATION  SETUP  AND  PARAMETERS 

The  generation  of  the  scenario  depends  upon  the  selections  made  in  the  initial  set 
up  process  which  is  conducted  by  the  instructor  in  this  experiment.  Once  the  selections 
have  been  made  the  instructor  is  responsible  monitoring  the  simulation  and  providing  the 
keyboard  inputs  that  will  drive  the  AI  of  the  system.  The  instructor’s  involvement  will  be 
discussed  later  in  this  chapter  in  detail. 


25 


1.  Scenario  Generation 

As  discussed  in  Chapter  IV,  the  scenario  chosen  for  this  experiment  is  the 
potential  threat  of  conflict  with  China  if  a  hostile  situation  develops  between  China  and 
Taiwan.  The  territory  directly  involved  within  the  scenario  will  be  the  body  of  water 
called  the  Strait  of  Taiwan,  as  shown  in  Figure  9. 


The  graphical  user  interface  provided  by  this  simulation  starts  off  with  the  user 
choosing  what  region  of  the  world  they  would  like  to  operate  in.  The  setup  lists  other 
areas  of  the  world;  however,  only  the  Strait  of  Taiwan  will  be  active  in  this  thesis.  In 
conjunction  with  selecting  the  region  of  interest  there  will  be  other  options  as  discussed 
in  chapter  IV,  such  as,  time  of  day,  and  intensity  level.  All  of  these  selections  shown  in 
the  setup  will  display  various  options,  but  again,  only  the  selection  needed  for  the 
scenario  will  be  active.  This  will  allow  for  future  implementation  and  provide  the 
illusion  of  a  complete  training  system. 
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Figure  9. 


The  scenario  was  conducted  in  two  versions  depending  on  the  stage  of  the 
experiment.  The  first  version  was  the  fixed  response  version;  in  other  words,  no  AI 
involved  and  actions  of  the  enemy  will  be  simulated  in  pre-determined  time  increments. 
A  story  board,  shown  in  Table  1,  was  created  that  illustrates  the  complete  scenario  in  a 
table  format  that  lists  the  time,  event,  expected  action,  and  the  contingent  action  of  the 
simulation.  Again,  this  version  will  be  time  driven  and  will  have  no  independent 
reactions  other  than  what  is  scripted. 


Time 

in 

Event 

Expected  Action 

Contingent  Action 

00:00 

Commence  Transit 

00:01 

News  Input 

News  reports  that  Taiwan 
abandons  “one-China”  principle 
but  does  not  declare 
independence.  China  removes 
rights  for  US  port  calls. 

Instructor  make  note 
of  reaction. 

00:02 

Report  of  Snoop  Tray-2 
radar  from  EP-3  on  patrol 
over  Strait  at  location 
south  of  Taiwan 

Match  radar  to  platform  (Kilo) 
Ask  for  HUMINT,  SIGINT,  and 
imagery  operators  for  any 
information  on  Chinese  Kilos. 

Instructor  make  note. 

00:03 

Recommend  vectoring  P-3 
supporting  ESG  transit  to  Kilo 
position  for  localization 

Kilo  will  be  located 
within  its  normal 
range  at  T  +01:00 

00:04 

Watchstander  should  recognize 
that  Chinese  Kilos  carry  shoulder 
launched  SAM’s  with  range  of  X 
NM  and  inform  watch  captain. 
Watchstander  should  recommend 
to  watch  captain  that  P-3  maintain 
safe  standoff  distance  via  USWC. 

Receive  report  that 
Snoop  Tray-2  Ceases 
Emitting  and  receive 
report  of  Chinese 
submarine  exercise 
ongoing. 

00:05 

Report  of  local  Taiwanese 
fishing  vessels  being 
harassed  off  coast  of 
Taiwan  by  Chinese 
frigates 

Watchstander  should  be 
constantly  monitoring  all  contacts 
on  display.  If  unknown  contacts, 
should  be  contacting  other  ships 
near  contact’s  location  for  contact 
information. 

Instructor  make  note. 

00:06 

Watchstander  should  locate 
Chinese  frigates  on  display;  if 
not,  trying  to  correlate  with 
contacts  on  display  by  contacting 
other  units  in  area  for  their 
INTEL. 

Chinese  Frigate  will 
be  shown  changing 
course  in  opposite 
direction  and 
becoming  a  non  threat. 

00:07 

Unknown  surface  contact 
heading  import. 

Watchstander  should  be  interested 
in  surface  contact  and 
identification. 

P-3  will  report  that  it 
is  a  Commercial 
Tanker  heading  in 
port,  (will  longer  be  an 
issue) 
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Time 

in 

Event 

Expected  Action 

Contingent  Action 

00:08 

Unknown  radar  contact 
off  the  North  tip  of 
Taiwan  at  speed  of  X 
knots  heading  toward  the 
capital  of  Taiwan 
(Taipei). 

Watchstander  should  have  noticed 
the  unknown  contact  was  an 
aircraft  from  speed  and  already 
made  attempt  to  identify  through 
means  available  to  him/her. 

CG  will  report  that  it 
shows  IFF  of  a 
Chinese  J-6  and  they 
are  conducting  a 
query. 

00:09 

Watchstander  should  report 
weapons  capability  of  J-6  and 
recommend  to  watch  captain  to 
label  contact  as  hostile. 

CG  will  report  that 
query  of  aircraft  was 
ignored  and  they  are 
heading  in  direction  of 
CG  off  north  coast  of 
Taiwan. 

Watchstander  should  recommend 
diverting  CAP  in  direction  of 
aircraft  for  a  visual  and  request  of 
intentions  and  change  of  course. 

Instructor  make  note. 

00:10 

J-6  will  continue  to  get 
closer  to  CG  and  emits 
fire  control  radar. 

00:10 

Scenario  Ends. 

Table  1.  Story  board  for  scripted  version 


The  second  version  is  supported  by  AI  and  the  simulation  will  progress  with  the 
keyboard  inputs  of  the  instructor  dependent  on  the  response  of  the  user.  Similar  to  the 
fixed-version,  there  is  a  story  board,  shown  in  Table  2;  however,  the  story  board  involved 
with  the  AI  version  tells  the  instructor  with  what  key  to  push  (1,  2,  or  3)  during  certain 
time  blocks  in  response  to  the  user’s  actions.  The  total  time  for  the  AI  version  will  be  the 
same  as  the  fixed-version,  10  minutes. 


Time 
in  (HH: 
MM) 

Event 

Expected  Action 

Key 

Contingent  Action 

00:00 

Commence 

Transit 

00:01 

News  input 

News  reports  that  Taiwan 
abandon’s  “one-China” 
principle  but  does  not  declare 
independence.  China  removes 
rights  for  US  port  calls. 

1 

If  threat  to  level  of  transit  is 
noted,  scenario  will  continue  as 
normal.  (Note  for  instructor) 
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Time 
in  (HH: 
MM) 

Event 

Expected  Action 

Key 

Contingent  Action 

2 

Instructor  dependent:  Moderate 
Level  of  preferences. 

3 

If  this  is  not  noted  by  the  watch 
standers  as  an  increased  threat 
level  of  transit,  then  scenario 
should  become  more  difficult. 
(Note  for  instructor) 

00:02 

Report  of 
Snoop  Tray-2 
Radar  from 
EP-3  on  patrol 
over  Strait  at 
location  south 
of  Taiwan 

Match  radar  to  platform  (Kilo). 
Ask  for  HUMINT,  SIGINT, 
and  imagery  operators  for  any 
information  on  Chinese  Kilos. 

00:03 

Recommend  vectoring  P-3 
supporting  ESG  transit  to  Kilo 
position  for  localization 

1 

If  instant  reaction  is:  P-3  is 
vectored,  Kilo  will  be  located 
within  its  normal  range  at  T 
+01:00 

2 

Kilo  will  be  located  at  T 
+02:00. 

3 

If  P-3  is  not  vectored,  Kilo  will 
disappear  from  radar  and 
location  will  be  unknown 
leaving  ESG  vulnerable. 

00:04 

Watchstander  should  recognize 
that  Chinese  Kilos  carry 
shoulder  launched  SAM’s  with 
range  of  X  NM  and  inform 
watch  captain.  Watchstander 
should  recommend  to  watch 
captain  that  P-3  maintain  safe 
standoff  distance  via  USWC. 

1 

If  so,  receive  report  that  Snoop 
Tray-2  ceases  emitting  and 
receive  report  of  Chinese 
submarine  exercise  ongoing. 

2 

Receive  report  that  Chinese 
submarine  exercise  ceased  and 
has  intended  movement 
towards  ESG. 

3 

If  not,  P-3  might  be  engaged  by 
shoulder  launched  SAM  and 
receive  HUMINT  relay  report 
of  Kilo-366  being  loaded  with 
torpedoes  6  days  ago  before 
departing  homeport. 

00:05 

Report  of  local 
Taiwanese 
fishing  vessels 
being  harassed 
off  coast  of 
Taiwan  by 
Chinese 
frigates 

Watchstander  should  be 
constantly  monitoring  all 
contacts  on  display.  If 
unknown  contacts,  should  be 
contacting  other  ships  near 
contact  for  contact  information. 

1 

If  this  happens,  Chinese  frigate 
will  be  shown  changing  course 
in  opposite  direction  and 
becoming  a  non  threat. 
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Time 
in  (HH: 
MM) 

Event 

Expected  Action 

Key 

Contingent  Action 

2 

Continued  harassment  of 
fishing  vessel  will  lead  to 
bridge-to-bridge  transmission 
of  SOS.  And  frigate  will  cease 
harassment. 

Watchstander  should  locate 
Chinese  frigates  on  display,  if 
not,  trying  to  correlate  with 
contacts  on  display  by 
contacting  other  units  in  area 
for  their  INTEL. 

3 

If  this  does  not  happen,  Chinese 
frigate  will  continue  to  harass 
and  Taiwanese  response  will 
elevate  and  could  lead  to  shots 
fired  toward  fishing  vessel. 

00:06 

Unknown 

Surface 

contact 

heading 

import. 

Watchstander  should  be 
interested  in  surface  contact 
and  identification. 

1 

If  user  asks  about  the  unknown 
surface  contact,  P-3  will  report 
that  it  is  a  commercial  tanker 
heading  in  port,  (will  no  longer 
be  an  issue) 

2 

Message  that  Intel  received  on 
certain  shipping  vessels 
carrying  biological  agents 
onboard  and  to  keep  look  out. 

3 

If  not,  receive  report  of 
mobilization  of  PLA  in 
preparation  for  attack  of 
Taiwan. 

00:07 

Unknown 
radar  contact 
off  the  North 
tip  of  Taiwan 
at  speed  of  X 
knots  heading 
toward  the 
capital  of 
Taiwan 
(Taipei). 

Watchstander  should  have 
noticed  the  unknown  contact 
was  an  aircraft  from  speed  and 
already  made  attempt  to 
identify  through  means 
available  to  him/her. 

1 

If  so,  CG  will  report  that  it 
shows  IFF  of  a  Chinese  J-6  and 
query  in  progress. 

2 

If  noticed  and  did  nothing  to 
find  out,  CG  will  report  that 
query  being  conducted  and 
response  ongoing. 

3 

If  not,  CG  will  report  that  query 
of  aircraft  was  ignored  and  they 
are  heading  in  direction  of  CG 
off  north  coast  of  Taiwan. 

Watchstander  should  report 
weapons  capability  of  J-6  and 
recommend  to  the  watch 
captain  to  label  contact  as 
hostile. 

1 

CG  will  report  query  successful 
and  J-6  changing  course. 

2 

CG  will  conduct  a  2nd  Query 
and  warn  J-6  that  they  will 
shoot  to  kill. 

3 

J-6  will  ignore  query  and 
continue  heading  towards  CG. 
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Time 
in  (HH: 
MM) 

Event 

Expected  Action 

Key 

Contingent  Action 

Watchstander  should 
recommend  diverting  CAP  in 
direction  of  aircraft  for  a  visual 
and  request  of  intentions  and 
change  of  course. 

1 

If  so,  J-6  will  change  directions 
and  head  back  in  the  opposite 
direction. 

2 

If  suggest  something  else,  J-6 
will  come  within  1 0  nm  and 
turn  around.  CG  will  report 
this. 

3 

If  not,  J-6  will  continue  to  get 
closer  to  CG  and  emits  fire 
control  radar. 

00:10 

Scenario  Ends. 

Table  2.  Story  board  for  AI  version 
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VI.  EXPERIMENT 


A.  EXPERIMENTAL  SETUP  TO  VALIDATE  THE  USE  OF  THE  SYSTEM 

This  chapter  will  describe  the  experimental  design  and  subjects  used  to  conduct 
the  experiment. 

1.  Subjects  Used  to  Conduct  Experiment 

The  experiment  was  conducted  using  students  qualified  in  Naval  Surface  Warfare 
from  the  Naval  Postgraduate  School  in  Monterey,  California.  It  is  essential  that  the 
subjects  have  the  basic  knowledge  and  understanding  of  how  the  Global  Command  and 
Control  (GCCS)  display  functions,  experience  in  dealing  with  contacts  of  various  entities 
and  the  threat  of  proximity  in  international  waters,  as  well  as  rules  of  engagement  (ROE) 
in  a  potential  combat  zone. 

This  experience  comes  from  standing  watch  while  underway  onboard  a  combat 
capable  ship  and  obtaining  either  the  Combat  Information  Center  Watch  Officer 
(CICWO)  or  Tactical  Actions  Officer  (TAO)  qualification.  The  requirement  for  the 
qualification  in  Surface  Warfare  for  this  experiment  fulfills  the  basic  knowledge 
constraint,  because  CICWO  is  a  prerequisite  to  be  Surface  Warfare  qualified. 

2.  Experimental  Setup 

The  experiments  were  conducted  at  the  Naval  Postgraduate  School  in  which 
student  volunteers,  qualified  in  Surface  Warfare,  were  used  to  determine  which  of  the 
two  systems  used  in  this  experiment  were  superior.  First,  the  subjects  were  informed  that 
the  training  simulation  was  a  simple  reconstruction  of  a  current  system  used  at  FTCPAC 
that  included  the  basic  functionalities  and  are  limited  to  one  scenario  each  for 
experimental  purposes.  Additionally,  they  were  instructed  in  how  to  conduct  various 
tasks  within  the  GUI  and  what  they  were  expected  to  accomplish  within  the  training 
simulation. 

The  experiment  was  conducted  in  two  groups:  1)  control  group  and  2) 
experimental  group.  The  control  group  was  made  up  of  subjects  who  operated  the  fixed- 
version  training  simulation  and  the  experimental  group  subjects  were  operating  the  AI- 
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version  training  simulation.  Subjects  were  randomly  assigned  to  the  two  groups  by 
choosing  every  other  one  based  on  their  arrival  time. 

Control  group:  conducted  a  total  of  three  simulation  runs  of  the  fixed-version 
simulation. 

Experimental  group:  conducted  a  total  of  three  simulation  runs  of  the  AI- version 
simulation. 

At  the  end  of  the  test  runs,  there  was  a  face-to-face  debrief,  which  included  a 
questionnaire  for  each  group  that  was  used  afterward  for  comparison  analysis.  The 
questionnaire  addressed  questions  for  each  event  of  the  simulation,  separately,  in  order  to 
determine  how  many  predictions  a  subject  anticipated  for  the  enemy’s  reactions  within 
that  particular  event.  (See  Appendix  A  for  the  questionnaire  developed  for  this 
experiment) 

3.  Experimental  Design 

The  purpose  of  this  experiment  is  to  demonstrate  the  importance  of  applying  AI 
technology  to  current  training  simulations.  The  system  being  enhanced  currently  uses 
scripted  events  that  carry  out  enemy  actions  independent  of  the  user’s  inputs/reactions. 
This  is  extremely  detrimental  to  overall  training  because  it  does  not  take  into  account  the 
variability  of  the  real  world.  After  the  first  run  of  a  particular  scenario,  every  run 
thereafter  is  one-hundred  percent  predictable  because  the  enemy  will  perform  the  same 
actions  every  time.  In  other  words,  the  subject  is  expected  to  predict  only  one  enemy 
response  per  event  with  the  scripted  version.  Ironically,  the  current  system  does  produce 
overall  performance  improvement  between  scenario  runs;  however,  this  is  only  because 
the  user  knows  exactly  what  is  going  to  happen. 

The  use  of  AI  technology  not  only  will  keep  the  user  engaged,  but  enhance 
overall  combat  readiness  (Boldo87).  This  is  of  course  important  to  any  military  branch, 
for  the  threats  in  the  world  today  are  much  different  than  those  a  generation  prior.  The 
military  today  is  dealing  with  threats  that  are  non-conventional  and  unpredictable  in  a 
sense  that  we  are  operating  in  environments  that  are  not  familiar  (e.g.,  urban  warfare  or 
religious  persecution). 
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The  subjects  were  not  told  what  system  they  will  be  assigned  to,  only  that  they  are 
either  in  system  A  or  system  B.  A  scripted  format  of  steps  to  be  taken  was  strictly 
followed  in  order  to  ensure  consistency  throughout  the  experiment.  (See  Appendix  B) 

The  variable  to  be  analyzed  was  the  number  of  predictions  for  the  enemy’s 
response  within  each  event,  indicated  by  the  user  in  the  questionnaire. 

The  hypothesis  of  the  experiment  is:  Adding  AI  to  a  training  simulation  causes 
the  subject  to  consider  more  possible  enemy  responses  per  event  throughout  multiple  runs 
of  the  simulation. 

B.  ANALYSIS 

Data  obtained  from  this  experiment  is  the  number  of  enemy  response  predictions 
per  event.  Each  subject  answered  a  questionnaire  which  lists  each  event  separately 
requiring  a  number  that  describes  how  many  of  the  enemy’s  responses  they  anticipated 
during  that  event  given  the  history  of  the  last  three  runs  of  the  simulation.  In  other 
words,  if  the  subject  ran  a  fourth  experiment,  what  they  would  expect  to  be  the  enemy’s 
response  based  off  of  what  had  happened  in  the  last  three  simulation  runs. 

The  control  group  responded  as  expected  given  the  exact  same  scenario  three 
times  in  a  row.  Because  the  enemy  responded  the  same  all  three  runs,  they  anticipated 
that  the  fourth  run  would  be  the  same.  The  results  are  shown  in  Table  3. 


Fixed  Version 


□  Average 
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Table  3.  Distribution  of  Average  Number  of  Anticipated  Responses  for  Control  Group 
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c. 


RESULTS  OF  EXPERIMENT 

Is  hypothesis  correct?  Adding  AI  to  a  training  simulation  caused  the  subject(s)  to 


consider  more  possible  enemy  responses  per  event  on  the  final  “Test”  run  (See  Tables  4, 
5,6). 


AI  Version 


Events 


Table  4.  Distribution  of  Average  Number  of  Anticipated  Responses  for  Experimental 

Group 


Fixed  Version 

AI  version 

Event 

Average  Number  of 
anticipated  Enemy 
response(s) 

Average  Number  of 
anticipated  Enemy 
response(s) 

Event  1 

1.0 

2.5 

Event  2 

1.0 

3.75 

Event  3 

1.0 

3.75 

Event  4 

1.0 

3 

Event  5 

1.0 

3.25 

Event  6 

1.0 

2.5 

Event  7 

1.0 

2.5 

Table  5.  Table  of  Data  Obtained  from  Experiment 
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Distribution  of  Anticipated  Enemy  Responses 
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Table  6.  Distribution  of  Anticipated  Enemy  Responses,  Displaying  Control  Group  vs. 

Experimental  Group 


However,  while  we  did  not  perform  a  statistical  test,  the  data  makes  clear  that  the 
null  hypothesis  (that  the  responses  of  control  and  experimental  groups  are  from  the  same 
distribution)  must  be  rejected,  since  there  is  almost  no  overlap  between  the  two  response 
sets.  The  control  group  uniformly  responded  with  “1”  for  the  number  of  possible  enemy 
responses.  In  the  experimental  group,  only  one  individual  in  one  instance  gave  the 
response  “1”.  See  complete  data  from  the  experiment  in  Appendix  C. 

1.  Analysis  of  Results 

Common  sense  tells  us  that  the  results  of  conducting  an  experiment  with  this 
setup  will  result  in  more  predictions  of  anticipated  enemy  responses  in  the  AI  version 
versus  the  fixed  version  training  system.  However,  this  is  dependent  on  the  setup  and 
consistency  in  which  the  experiment  is  presented  to  the  subjects. 

This  approach  to  enhancing  legacy  military  training  systems  by  incorporating  AI 
into  the  already  existing  system  is  very  simple  and  will  ensure  that  the  subjects  are  being 
presented  with  an  environment  that  is  more  realistic. 

When  subjects  use  a  training  tool  whose  objective  is  to  increase  combat  readiness, 
how  combat  ready  is  a  person  whose  only  reference  to  combat  is  a  training  tool  that  is 
one  hundred  percent  predictable? 
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2.  Control  Group 

All  subjects  who  were  in  the  control  group  realized  that  the  enemy  responses  were 
fixed  during  the  second  run  of  the  simulation.  Of  the  four  subjects  who  conducted  the 
controlled  experiment,  the  first  three  verbally  noted  that  “the  enemy  did  the  same  thing 
last  time ”,  when  the  third  event  occurred  in  the  second  simulation  run.  While  the  fourth 
subject  realized  the  enemy  had  the  same  reaction  during  the  fourth  event  of  the  second 
simulation  run.  Upon  realizing  that  the  enemy’s  actions  were  going  to  be  the  same 
regardless  of  what  they  did,  the  subjects  were  not  as  observant  as  in  the  beginning  and  all 
pursued  some  sort  of  conversation  outside  the  area  of  the  experiment. 

Results  of  the  control  group  confirmed  the  complacency  assumption  of  scripted 
events.  Basic  human  nature  applied  and  subjects  were  not  as  observant  of  the  scenario 
and  its  components  when  they  came  to  understand  that  they  did  not  control  or  effect  the 
enemy’s  reactions. 

3.  Experimental  Group 

Contrary  to  the  control  group,  the  subjects  in  the  experimental  group  understood 
that  their  actions  (verbally)  decided  the  actions  of  the  enemy.  Two  subjects  noted  that  the 
key  strokes  made  during  each  event  made  them  believe  that  their  responses  were  being 
addressed  and  that  the  enemy  was  reacting  because  of  the  key  strokes.  Whether  or  not 
this  was  the  case,  the  key  strokes  played  a  essential  role  in  keeping  the  subjects  observant 
during  the  simulation  runs. 

During  the  second  run  of  the  simulation  all  subjects  understood  that  the  enemy 
was  reacting  differently  than  the  first  simulation  run.  This  was  mainly  due  to  the  fact  that 
during  the  first  simulation  run,  every  subject  had  a  high  intensity  reaction  from  the 
enemy.  This  increased  intensity  level  was  set  because  each  subject  was  getting  use  to  the 
controls  and  functionality  of  the  simulation  during  the  first  run  and  most  tended  not  to  be 
as  verbal  as  needed  by  the  experiment.  Once  each  subject  was  settled  into  their  own 
routine  the  second  and  third  run  went  more  smoothly  with  less  and  less  intense  reactions 
from  the  enemy. 
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4.  Other  Results 

Overall  the  simulation  runs  of  each  group  went  smoothly  and  as  perceived. 
However,  one  subject  did  not  understand  what  was  needed  from  him  in  the  debrief 
questionnaire  and  because  too  much  information  was  given  to  him,  the  answers  received 
by  this  subject  had  to  be  thrown  out. 
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VII.  SUMMARY,  CONCLUSIONS,  AND  FUTURE  WORK 


A.  SUMMARY 

In  the  real-world,  the  enemy  will  not  always  react  in  the  same  way  when 
presented  with  an  identical  situation  or  environment.  So  why  should  we  be  training  our 
military  personnel  on  a  system  that  uses  this  antiquated  approach. 

Similar  to  the  ‘Choose  your  own  adventure’  books,  a  series  of  children’s  books 
first  published  by  Bantam  Books  from  1979  to  1998,  the  framework  of  this  system 
requires  numerous  potential  responses  in  the  database.  This  will  ensure  that  the  user’s 
actions  will  result  in  a  different  enemy  action  each  time  the  simulation  is  played  at  least 
more  than  four  runs  consecutively.  The  number  of  available  enemy  responses  will  be 
dependent  on  the  system  design  and  intention  of  training. 

In  a  paper  written  by  Francis  Kapper  in  1981,  he  summarizes  proper  use  of 
military  simulations  and  the  potential  level  of  validity  achievable  by  those  simulations: 
“The  most  appropriate  and  valid  objectives  for  using  war  games  and  simulations  within 
the  DoD  context  are  to:  better  understand  complex  phenomena,  identify  problems, 
evaluate  alternatives,  gain  new  insights,  and  broaden  one’s  perspectives”  (Kapp81).  The 
variability  of  the  enemy’s  responses  to  the  user’s  actions  was  accomplished  with  a  very 
simple  technique  allowing  the  enemy  to  have  more  than  one  response.  The  three  to  four 
responses  available  to  choose  from  dependent  on  the  users  actions  was  enough  to  ensure 
the  subject  was  engaged  and  maintained  continuous  observation  of  his/her  simulation 
environment.  As  mentioned  earlier,  scripted  game -based  environments  tend  to  lose  the 
attention  of  users  once  they  realize  no  matter  what  they  do,  the  enemy  will  always  react 
the  same. 

B.  CONCLUSIONS 

In  recent  years  there  has  been  a  dramatic  interest  in  developing  simulations  and 
virtual  environments  for  training  purposes,  particularly  in  the  military.  The  demands  on 
military  simulations  are  continually  increasing,  primarily  motivated  by  fiscal  necessity 
and  secondly,  due  to  the  pressure  to  employ  such  simulations,  which  has  led  to 
exploration  of  new  methods  for  modeling  combat  activities.  Billons  of  dollars  are  spent 
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producing  simulations  for  combat  training;  unfortunately,  little  is  spent  on  validating  the 
products  and  even  fewer  spent  on  the  effects  of  utilizing  artificial  intelligence  (AI)  to 
enhance  legacy  military  training  systems. 

The  purpose  of  conducting  this  experiment  in  such  a  manner  as  described  above  is 
to  demonstrate  the  importance  of  applying  AI  technology  to  current  combat-related 
training  systems,  despite  the  simplicity  of  the  AI.  This  work  demonstrates  that  simple  AI 
is  better  than  no  AI. 

Again,  most  legacy  combat-related  training  tools  use  scripted  events  that  carry  out 
enemy  actions  dependent  of  the  user’s  inputs/actions.  This  is  extremely  detrimental  to 
overall  training  because  it  does  not  take  into  account  the  variability  of  the  real  world. 
After  the  first  run  of  a  particular  scenario  in  a  scripted  system,  every  run  thereafter  is  one- 
hundred  percent  predictable  because  the  enemy  will  react  the  same  way  every  time.  In 
other  words,  the  subject  is  expected  to  predict  only  one  enemy  response  per  event  with 
the  scripted  version.  Ironically,  these  systems  will  produce  overall,  performance  growth; 
however,  this  is  only  because  the  user  knows  exactly  what  the  enemy  will  do. 

C.  FUTURE  WORK 

A  recommended  extension  of  the  experiment  would  be  to  execute  a  total  of  four, 
simulation  runs  with  subjects  in  both  the  control  and  experimental  group.  The  fourth 
simulation  run  would  be  carried  out  with  a  completely  different  scenario  utilizing  AI  for 
both  groups.  The  reaction  time  of  each  user  could  be  recorded  to  determine  performance 
levels  based  on  the  experience  obtained  during  the  first  three  runs  within  their  own  group. 
This  could  establish  the  application  of  AI  in  determining  the  efficiency  of  training  when 
presented  with  a  completely  new  scenario. 

Further  research  and  development  of  techniques  in  application  of  AI  in  combat 
training  tools  is  highly  encouraged,  for  the  reason  that  it  can  greatly  contribute  to  the 
combat  fighting  proficiency  of  our  future  Sailors.  In  particular,  in  order  create  a  more 
useful  training  tool  for  military  intelligence  team  trainers;  AI  for  chat  capabilities  with 
other  entities  in  the  Battle  Group  should  be  developed. 


42 


APPENDIX  A.  DEBRIEF  QUESTIONNAIRE 


This  handout  was  used  in  order  to  obtain  the  data  needed  for  the  thesis  hypothesis. 
The  subject  was  told  to  answer  the  questions  with  respect  to  only  the  simulation  and  their 
experience  with  the  three  simulation  runs.  In  other  words,  if  the  subject  ran  a  fourth 
experiment,  what  they  would  expect  to  be  the  enemy’s  response  based  off  of  what  had 
happened  in  the  last  three  simulation  runs. 


Patricia  A.  Sweat 
MOVES 
Watkins  Hall 

Naval  Postgraduate  School 
Monterey,  California  93943 


831-583-0988 

541-771-2782 

pasweat@nps.edu 


DEBRIEF  QUESTIONNAIRE 

First  Event:  (Report  of  Snoop  Tray-2  Radar  from  EP-3  on  patrol  over  Strait  at 
location  south  of  Taiwan) 

How  many  reactions  of  the  enemy  did  you  anticipate  given  the  current  situation? 

I  I  Once  P-3  vectored,  Kilo  located  instantly 
I  I  Kilo’s  exact  location  could  not  be  determined. 

I  I  Kilo  not  located  and  datum  disappears  from  radar,  leaving  BG  and  ESG 
vulnerable. 

UlList  other  reactions:: _ 


Second  Event:  (Chinese  Kilo  threat  determination) 

How  many  reactions  of  the  enemy  did  you  anticipate  given  the  current  situation? 
I  I  Receive  report  of  Chinese  submarine  exercises  ongoing. 
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I  I  Receive  report  of  Chinese  submarine  intended  movement  towards  area  of 
Battle  group  or  ESG. 

I  I  P-3  might  be  engaged  by  shoulder  launched  SAM 

I  I  Receive  report  of  Kilo-366  being  loaded  with  torpedoes  prior  to  departing 
homeport. 

I  I  List  other  reactions: _ 

Third  Event:  (Report  of  Local  Taiwanese  Fishing  vessels  being  harassed  off 
coast  of  Taiwan  from  Chinese  Frigates) 

How  many  reactions  of  the  enemy  did  you  anticipate  given  the  current  situation? 

□  Chinese  Frigate  change  course  and  stop  harassment. 

I  I  Continued  harassment  from  Chinese. 

I  I  Call  for  help  from  fishing  vessels  over  Bridge-to-Bridge. 

I  I  Continued  harassment  of  fishing  vessels  lead  to  elevation  to  Taiwanese 
response. 

I  I  List  other  reactions: _ 

Fourth  Event:  (Unknown  Surface  contact  heading  import) 

How  many  reactions  of  the  enemy  did  you  anticipate  given  the  current  situation? 

I  I  Identified  as  Chinese  Naval  Vessels. 

I  I  Receive  report  that  it’s  a  commercial  shipping  vessel. 

I  I  Receive  report  of  commercial  vessels  carrying  biological  agents  onboard. 

I  I  No  response  from  surface  vessel  leading  to  concern  of  threat  to  Taiwan. 

□  List  other  reactions: _ 
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Fifth  Event:  (Unknown  radar  contact  off  the  North  Tip  of  Taiwan  at  speed  of  X 
knots  heading  toward  the  capital  of  Taiwan  -  Taipei). 

How  many  reactions  of  the  enemy  did  you  anticipate  given  the  current  situation? 

I  I  Report  of  IFF  and  contact  identified. 

I  I  Report  that  contact  Chinese  aircraft  and  query  in  progress. 

I  I  Report  that  query  is  in  progress  with  contact  and  no  response. 

I  I  Air  contact  heading  towards  ESG. 

I  I  List  other  reactions: _ 

Sixth  Event:  (Query  of  Air  contact  in  progress) 

How  many  reactions  of  the  enemy  did  you  anticipate  given  the  current  situation? 

I  I  Report  that  query  successful  and  aircraft  changing  course. 

I  I  Report  first  query  successful  however,  aircraft  not  changing  course  and 
second  query  ongoing  questioning  intentions. 

I  I  Report  that  queries  ignored  and  preparing  for  Battle  Stations. 

I  I  List  other  reactions: _ 

Seventh  Event:  (Air  craft  movement  in  progress) 

How  many  reactions  of  the  enemy  did  you  anticipate  given  the  current  situation? 

I  I  Aircraft  will  change  course  and  head  in  opposite  direction. 

I  I  Aircraft  will  continue  on  course  towards  Battle  Group  getting  close  enough  to 
upset,  but  turn  around  in  time  not  to  be  shot  at. 

I  I  Aircraft  will  continue  on  course  towards  Battle  Group  and  emits  fire  control 
radar. 
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I  I  List  other  reactions: _ 

During  any  time  in  the  experiment  did  you  believe  that  there  was  a  sense  of 
variability  with  the  enemy’s  actions?  (i.e.  did  the  enemy  react  in  response  to  your 
actions,  or  did  the  enemy  react  regardless  of  what  you  chose  to  do  or  say).  Briefly 
explain  your  answer. 

□  Yes 

□  No 

Any  comments  or  recommendations. 


Thank  you  for  your  time,  have  a  great  day! 
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APPENDIX  B.  EXPERIMENTAL  PROCEDURE 


Note:  The  quoted  paragraphs  are  what  was  said  verbatim  in  order  to  maintain 
consistency  throughout  the  experiment. 

“I  will  be  reading  from  a  script  in  order  to  maintain  consistency  among  the 
participants  and  to  ensure  that  no  detail  is  left  out  which  might  be  devastating  to  the 
overall  results  of  this  experiment.” 

1 .  Subject  will  walk  in  and  sit  down. 

a.  Subject  will  be  given  the  Participant  Consent  Form  and  Minimal  Risk 
Statement  for  signature 

“First,  I  will  be  giving  you  a  Participant  Consent  Form  &  Minimal  Risk  Statement 
for  signature  and  for  IRB  and  my  records.  The  purpose  of  this  experimental  study  will  be 
used  to  improve  the  use  of  advanced  techniques  to  enhance  future  military  training 
simulations.  Certain  details  will  be  left  out  in  order  not  to  affect  the  experiment, 
however,  please  feel  free  to  ask  after  the  experiment  is  completed.  There  will  be  three 
phases  in  this  experiment:  1)  this  5  minute  brief  which  will  be  used  to  familiarize  you 
with  the  training  system  2)  training  and  execution  phases  lasting  approximately  5  minutes 
per  simulation,  during  which  you  will  be  expected  to  accomplish  a  number  of  tasks 
related  to  identifying,  interpreting  data,  and  making  recommendations  based  on  data 
collected,  and  3)  5  minute  debrief  which  will  consist  of  a  questionnaire.” 

Second,  I  need  to  ensure  that  you  understand  that  this  project  does  not  involve  any 
reasonably  foreseeable  risks  or  hazards  greater  than  those  encountered  in  everyday  life. 

Additionally,  I  must  tell  you  that  all  recodes  of  this  study  will  be  kept  confidential 
and  that  your  privacy  will  be  safeguarded.  No  information  will  be  publicly  accessible 
which  could  identify  you  as  a  participant,  and  you  will  be  identified  only  as  a  code 
number  on  all  research  forms. 

All  records  of  your  participation  will  be  maintained  by  NPS  for  five  years,  after 
which  they  will  be  destroyed.  Your  participation  is  strictly  voluntary,  and  if  you  agree  to 
participate,  you  are  free  to  withdraw  at  any  time  without  prejudice.” 


47 


b.  Subject  will  sign  consent  form. 

2.  Subject  will  be  briefed  on  what  the  current  trainer  at  FITCPAC  accomplishes. 

a.  Trainer  for  Carrier  BG  Intelligence  cells. 

b.  Assists  FITCPAC  instructors  in  analysis  of  individual  interpretations 
of  intelligence  information. 

3.  Subject  will  be  introduced  to  the  functionality  of  the  simulation  provided  by 
the  GUI. 

a.  Subject  will  be  given  visual  aid  document  with  useful  symbology  for 
reference  through  out  the  simulation  runs. 

b.  What  each  symbol  represents  (i.e.  aircraft,  surface,  sub-surface, 
unknown,  friendly,  hostile. 

(Symbology  Sheet)  -  “This  sheet  should  be  used  as  a  reference  for  understanding 
the  symbology  used  in  this  experiment.  These  symbols  are  similar  to  those  seen 
in  a  Combat  Information  Center  on  ACDS  screen  and/or  on  GCCS  screen.  Feel 
free  to  refer  to  this  sheet  throughout  the  experiment.” 

c.  How  to  use  the  message  traffic  box 

(Screen  Shot  Sheet)  -  “This  is  a  screen  shot  of  what  the  simulation  will  look  like 
during  the  experiment.  The  tabbed  marked  ‘Messages’  will  be  where  you  check 
messages  that  are  generated  that  contain  reports  from  various  Intelligence  sources. 
In  order  to  read  the  content  of  the  messages  you  need  to  click  once  on  each  line 
item  that  shows  up  at  specific  times  throughout  the  simulation. 

It’s  important  to  constantly  check  if  new  messages  have  arrived.  Messages 
contain  information  on  correlation  of  contacts  and  intelligence  that  will  help  you 
decided  what  to  do  next.” 

d.  How  to  identify  various  entities  on  the  chart  provided  by  the  GUI,  (i.e. 
the  GCCS  display  portion) 

(Screen  Shot  Sheet)  -  “This  is  a  screen  shot  of  what  is  the  initial  contact 

information.  As  you  can  see  there  are  some  contacts  that  are  labeled  ‘unknown’. 
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These  are  the  contacts  that  you  need  to  correlate  with  by  reading  the  message 
traffic  and  analyzing  the  reports  from  the  Intelligence  sources.” 

4.  Subject  will  be  told  what  is  expected  of  them  and  how  to  accomplish  the 
tasks;  via  voice  or  the  GUI. 

“During  the  experiment,  you  will  be  expected  to  verbalize  your  thoughts,  i.e. 
verbalize  your  explanation  for  your  decision-making.  What  you  wondering  about, 
what  you  think  certain  entities  are  or  could  be,  and  concerns  with  certain  entities 
and  there  location  in  regards  to  the  Battle  Group  or  ESG.” 

a.  Information  gathering  techniques  (i.e.  messages,  voice  comms) 

b.  Interpretation  of  data  and  how  to  make  recommendations  to  the  watch 
captain 

c.  How  voicing  opinion  and  recommendations  is  important  to  the 
progression  of  the  simulation 

5.  Once  subject  understands  the  tasks  involved,  the  simulation  will  start. 

a.  Subject  will  conduct  four  simulation  runs  overall 

b.  Each  run  will  be  approximately  5  minutes  each 

6.  Once  all  runs  of  the  simulation  is  over,  debrief  will  commence. 

a.  Subject  will  be  given  questionnaire. 

b.  Explain  the  questionnaire  to  the  subject  and  how  accurate  and  correct 
answers  are  important  to  the  analysis. 

“When  answering  this  questionnaire,  please  keep  in  mind  that  the  answers  should 
be  directing  related  to  the  simulation,  in  other  words,  what  the  simulation 
provided  to  you  as  a  user.  When  asked  how  many  reactions  of  the  enemy  did  you 
anticipate  given  the  current  event,  the  answer  should  be  what  the  simulation 
allowed  you  to  anticipate.” 
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APPENDIX  C.  EXPERIMENT  RESULTS  FOR  SCRIPTED 

VERSION 
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APPENDIX  D.  CONTACT  SYMBOLOGY  HANDOUT 


This  handout  was  used  as  a  reference  for  symbology  used  in  the  simulation.  Even 
though  all  subjects  were  students  at  Naval  Postgraduate  School  with  Combat  Information 
Center  experience;  certain  details  are  forgotten  after  being  away  from  the  environment 
for  a  long  period  of  time. 
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Unknown 
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APPENDIX  E.  SCREEN  SHOT  SIMULATION  HANDOUT 


This  handout  was  used  in  order  to  illustrate  what  the  simulation  would  look  like 
as  well  as  explain  the  functionality  of  the  message  traffic  and  how  to  access  the 
messages’  content  by  clicking  on  the  icon. 


0  Simulation 


0  Controls 


Time:  21  0001  50  Z  Jan  07 


0  Intelligence  Reports 


Messages  Chat  Contacts 

9  Messages 

- 

Q  210001Z 

Q  21 0002Z 

Q  210003Z 

Q  210004Z 

Q  210005Z 

Q  210005Z 

ZNYCCCCC 
P  21 0001 Z  MAR  06 

FM  N  AVLANTF fTC P AC  NORFOLK  VA//N3Q// 

TO  USS  SHIPtfJJJ// 

INFO  CINCLfliNTFLT  NORFOLK  VA0N50 
BT 

C  O  N  F I D  E  N T I AL//N02300// 

NEWS  REPORTS  THAT  TAIWAN  ABANDON’S  "ONE-CHINA" 
PRINCIPLE  BUT  DOES  NOT  DECLARE  INDEPENDENCE 
CHINA  REMOVES  RIGHTS  FOR  US  NAVAL  SHIP  PORT 
CALLS. 
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APPENDIX  F.  SCREEN  SHOT  MESSAGE  TRAFFIC  HANDOUT 


This  handout  was  used  to  illustrate  the  initial  contact  identification  correlated  with 
icons  shown  on  the  chart.  If  for  some  reason  the  user  forgot  what  certain  contacts  were, 
the  tab  marked  ‘Contacts’  was  what  needed  to  be  clicked  to  refresh  her/his 
memory. 
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APPENDIX  G.  EXPERIMENTAL  RESULTS  FOR  SCRIPTED 
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APPENDIX  H.  EXPERIMENTAL  RESULTS 
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APPENDIX  I.  SCREEN  SHOTS  OF  SIMULATION 
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